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Abstract Body 

Limit 4 pages single-spaced. 


Background / Context: 

Description of prior research and its intellectual context. 


Sample attrition in randomized controlled trials (RCTs) can lead to biased impact estimates. This 
is because the process by which individuals leave the study sample could be related to both 
treatment status and the outcomes of interest, leading to systematic differences between the 
individuals remaining in the treatment and control groups. While attrition might occur in a way 
that is unrelated to outcomes and treatment status, it is rarely possible to know this since the 
researcher typically does not control or understand the process by which individuals leave the 
study sample. Consequently, findings from RCTs that experience sample attrition should be 
regarded as less credible than findings from RCTs that experience no sample attrition. The fact 
that missing outcomes data can lead to bias in general (not just in RCTs) is a well- kn own issue 
that has received considerable attention in the methodological literature (Rubin 1976; Little and 
Rubin 2002; Puma et al 2009). 

To limit the potential for attrition bias to influence assessments of intervention effectiveness, 
several federally sponsored systematic evidence reviews have established a standard for 
acceptable levels of sample attrition in RCTs. These evidence reviews include the What Works 
Clearinghouse (WWC), the Home Visiting Evidence of Effectiveness Review (HomVEE), and 
the Teen Pregnancy Prevention Evidence Review (TPPER) [see http://ies.ed.gov/ncee/wwc/ , 
http://homvee.acf.hhs.gov/ , and http ://www.hhs . gov/ ash/oah/oah-initiatives/teen pregnancy/ db/1 . 

The attrition standard developed for the WWC was the progenitor of the other two standards. 
Initially the WWC attrition standard consisted of arbitrary cutoffs with respect to the overall 
attrition rate and the difference in attrition rates between the treatment and control groups. This 
initial standard had no empirical basis and no theoretical basis beyond the basic understanding 
that less attrition is likely preferable to more attrition. However, starting with version 2.0 of the 
WWC Procedures and Standards Handbook 

(http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=19) , the WWC attrition standard moved 
from being a set of arbitrary cutoffs to being a range of cutoffs based on a mathematical model of 
attrition bias with the choice of key parameters infonned by empirical data analysis. 


Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of this paper/presentation is to explain the WWC attrition model, the process for 
selecting key parameter values for that model, and how the model informed the development of 
the WWC attrition standard and can be used to develop attrition standards tailored to other 
substantive areas. 

Setting: 

Description of the research location. 
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(May not be applicable for Methods submissions) 

Not applicable 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

(May not be applicable for Methods submissions) 

Not applicable 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

(May not be applicable for Methods submissions) 

Not applicable 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

While others have developed approaches to bounding bias in a variety of contexts (for example, 
Manski 1990), we believe the WWC attrition standard may constitute the first use of model- 
based, empirically-supported bounds on attrition bias in the context of a federally-sponsored 
systematic evidence review. Previously federal agencies such as OMB and NCES have used 
standards for response rates in data collection efforts, but these standards are not based on 
estimates of expected bias, but rather on what response rates have been feasible to achieve in the 
past (http://www.whitehouse.gov/omb/inforeg/pmc survey guidance 2006.pdf and 
http://nces.ed.gov/pubs2003/2003601.pdf) . 

Meeting the WWC attrition standard (or one of the attrition standards based on the WWC 
standard) is now an important consideration for researchers conducting studies that could 
potentially be reviewed by the WWC (or other evidence reviews). 

Understanding the basis of this standard is valuable for anyone seeking to meet existing 
standards and for anyone interested in adopting this approach to developing a standard (that is, 
combining a theoretical model with empirical estimates of key parameters) in a new context. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 

When attrition is related to outcomes, different rates of attrition between the treatment and 
control groups can lead to biased impact estimates. Furthermore, if the relationship between 
attrition and outcomes differs between the treatment and control groups, then attrition can lead to 
bias even if the attrition rate is the same in both groups. The focus here is to specify a model 
showing how bias depends on the correlation between outcomes and attrition and the 
combination of overall and differential attrition in an RCT. 

To set up the model, consider a variable representing an individual’s latent (unobserved) 
propensity to respond, z. Assume z is normally distributed with mean zero and standard deviation 
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one. If the proportion of individuals who respond is P, an individual is a respondent if his or her 
value of z exceeds a threshold, z* : 


(1) z ><X> _1 (l-P) = z* 

where <E> is the standard normal cumulative distribution function. For example, in a scenario in 
which 75 percent of individuals respond (P = 0.75), an individual is a respondent if his or her 
value of z exceeds the value corresponding to the 25 th percentile in the z distribution [that is, 
exceeds O -1 (1-0.75)]. 

The outcome at follow-up, y, is the key variable of interest. We assume that y has a normal 
distribution. Moreover, we assume that y has mean zero and standard deviation one, given that 
any variable can be standardized in this way. The relationship between y and z can then be 
modeled as 


(2) y = a*z + u 

where a is the correlation between z and y, and it is a random variable that is independent of z. 1 
Note that there are no covariates and the model assumes no effect of the treatment on the mean 
outcome. If a is 1 or -1, the entire outcome is explained by the propensity to respond. If a is 
zero, none of the outcome is explained by the propensity to respond, which is the case when 
attrition is completely random. 

The correlation between the propensity to respond and outcomes may differ by treatment 
status. Therefore, we specify Equation (2) separately for treatment and control group members 
(subscripted by t and c, respectively): 


(3) y t =a*z t +u t 

y c = a c* z c+ u c- 

Because there is no true impact of the intervention on mean outcomes in this model, an 
unbiased estimator of the impact should, in expectation, find no difference in outcomes between 
the treatment and control groups. Therefore, in the presence of attrition, bias is equal to the 
difference between the expected values of y t and y c among respondents. Based on the 
properties of truncated nonnal distributions, the analytic fonnula for the bias, B, is 

(4) B = E(y t | z, > z*) — E(y c \ z c > z*) 

= a,E(z t \z, >z*)~ a c E{z c \ z c > z *) 

_ a,x ( p(0- l (\-P l )) a e x0(<S>-\l-P e )) 

P, P c 


1 In order for y to be a N(0,1) variable, u must be normally distributed with mean zero and standard deviation 
yjl — or • 
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where (f> is the standard nonnal density function. 

Equation (4) shows that bias can be generated by treatment-control differences in the 
response rates (P t and P ) or in the correlation between y and z (CX, and CX, ). If neither P nor a 

differs between these groups, then there is no bias because the same kinds of individuals respond 
from both groups. 2 However, if response rates differ between the treatment and control groups 
(P t ^ P ), then bias occurs even when a t = a c , because respondents in the treatment and control 
groups have different average values of z and, thus, different average values of y. Moreover, if 
a t # a c , then impact estimates will be biased even if the response rate is the same in both 
groups; respondents from the two groups will have different average values of y stemming from 
the differences in a. 3 


Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

The model specified above can be used to calculate the expected bias for any combination of 
overall and differential given estimates of GC t and CX, . With those estimates of bias, we can then 
construct bounds on acceptable rates of overall and differential attrition. Here we (1) describe the 
process for estimating a, and CX, and (2) illustrate attrition bounds corresponding to particular 

values of Cl., and CX c . 

1. Estimating (X t and a c 

If we were able to observe outcomes for both respondents and non-respondents in the 
treatment and control groups we could calculate CC t and 0C c directly. Let A„ denote the difference 

in outcomes, in effect size units, between respondents and nonrespondents in group g (either the 
treatment [/] or control [c] group). It can be shown that 


(5a) A = E(y s \ z > z*)-E(y s \ z < z*) = 


a g x^-\\-P g )) 


which implies that 


(5b) a g = 


^(l-zp). 


2 Those who attrite, nonetheless, will differ systematically from those who do not attrite, which may 
compromise the external validity of the study. However, we do not address that issue here. 

3 It is possible that a difference in the rate of attrition between groups could offset a difference between (X, and 
OC c . However we conservatively assume the opposite — that these differences are reinforcing, not offsetting. 
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Of course, we cannot observe outcomes for nonrespondents, so we cannot observe A„ 

directly. However, in studies that have both follow-up and baseline test scores, we can use the 
baseline test scores as proxies for the follow-up test scores because baseline scores are typically 
correlated with follow-up scores. Therefore we can use the difference in baseline test scores 
between respondents and nonrespondents as the proxy for A , and we use Equation (5b) to 

calculate a„ for g = t,c. 

2. Bounding Attrition Bias 

Given estimates of Cl, and 0C c , we can calculate expected bias for every combination of 

overall and differential attrition rates. We can then assess whether the expected bias for each 
combination of overall and differential attrition rates exceeds a maximum tolerable level (for the 
WWC, the level was set at 0.05 standard deviations). Figure 1 illustrates the combinations of 
overall and differential attrition that yield expected bias below the maximum tolerable amount 

for two different sets of assumptions for 0C t and OC c . The green/bottom-left region shows 
combinations of overall and differential attrition that yield attrition bias less than 0.05 under 
pessimistic assumptions ( (X, =0.45 and a c =0.39), the yellow/middle region shows additional 
combinations that yield attrition bias less than 0.05 under more optimistic assumptions ( 
(X, - 0.21 and <X C = 0.22), and the red/top-right region shows combinations that yield bias 
greater than 0.05 even under optimistic assumptions. 

Research Design: 

Description of the research design. 

(May not be applicable for Methods submissions) 

Not applicable 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

Not applicable 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

Not applicable 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

The model of attrition bias presented here, and the standard based on it, is an effort to protect 
consumers of research from drawing mistaken conclusions because of bias (in this case, attrition 
bias) in impact estimates. It is important to be clear about what this effort can and cannot do. 
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What we cannot do. We cannot claim that impact estimates from studies meeting the attrition 
standard are definitely free of attrition bias. Nor can we make precise probabilistic statements 
regarding the magnitude of attrition bias in studies that meet the attrition standard. This is 
primarily because (1) our empirical estimates of bias from past studies could be wrong (since we 
had to rely on pre-tests as a proxy for post-tests) and (2) our empirical estimates of bias from past 
studies may not apply to future studies (because, for example, future studies may involve 
different populations of students). 

What we can do. We can claim that impact estimates from studies meeting the attrition standard 
are likely to be less influenced by attrition bias than estimates from studies that do not. Also, this 
approach allows us to systematically manage the tradeoff between overall and differential 
attrition, recognizing that studies with low differential attrition are preferable to studies with high 
differential attrition, holding the overall rate constant. Finally, this approach allows us to reduce 
the influence of bias in impact estimates in a way that is systematic and informed by data, which 
we believe is preferable to arbitrary cutoffs. 
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Appendices 

Not included in page count. 
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Appendix B. Tables and Figures 

Not included in page count. 

FIGURE 1 

TRADEOFFS BETWEEN OVERALL AND DIFFERENTIAL ATTRITION 
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